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(S) Congestion avoidance in high-speed network carrying bursty traffic. 

(§7) A data communication network subject to bursty traffic employs a bandwidth allocation scheme to 
avoid congestion. When a source node has a burst of traffic to send, it first sends a bandwidth request 
message through the network from source to destination. At each intermediate node, this bandwidth 
request is examined and the node determines how much of the requested traffic tevel it wfll be able to 
support at a time in the future of one round-trip interval hence, and this node either grants the request 
or marks down the request to a level that it can support, then passes it on. When the request reaches the 
destination, it is returned along the same path to the source, and the source then employs the 
marked-down allocation to select the rate used to send the burst of data. The allocation for this source 
node remains in effect for a limited time, depending upon the amount of data to be sent in the burst, 
then returns to a "residual" level. 
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BACKGROUND OF THE INVENTION 

This invention generally relates to data communi- 
cation networks, and more particularly to congestion 
avoidance in networks having links exhibiting long s 
propagation delay and having bursty data flow. 

Broadband ISDN (integrated services digital net- 
work) systems are prone to severe buffer overflow 
problems at intermediate nodes. Data is thus lost and 
must be retransmitted, reducing the reliability and io 
capacity of the system. These losses are referred to 
as congestion loss, but this is not the result of an un- 
der-engineered network. Instead, this type of conges- 
tion is an inevitable result of the bursty nature of data 
in asynchronous (packetized) information transfer. 15 

In narrowband packet networks, feedback control 
mechanisms are able to manage the traffic load so 
that buffer overflow can be mostly avoided, or at least 
controllable. For example, networks based upon the 
so-called X-25 protocol provide two levels of flow con- 20 
trot; one controls all traffic across the physical link; 
and another layer cortrols traffic across one virtual cir- 
cuit Each layer's protocol provides both a window-ba- 
sed flow manager and a stop/go control mechanism. 
That is, a node having data to send is allocated a win- 25 
dow of fixed time, per unit time, and, in addition, the 
node can be shut off for a time period when capacity 
is all allocated. 

Connectionless networks, such as those using 
certain DECnet and DDN Internet Protocol, do not 30 
have positive controls as in XJ25, but still provide posi- 
tive feedback mechanisms. For example, "implicit" 
feedback mechanisms focus on sequence numbers in 
the packets; if a packet is dropped (as may be inferred 
from a gap in acknowledged sequence numbers) it 35 
may be determined that congestion is severe and so 
the sender drastically reduces its sending rate (as by 
reducing the window size). Or, "explicit" mechanisms 
provide warning of incipient congestion, so that sen- 
ders can usually reduce their rate before any packets 40 
are lost thus there is still feedback, but the data ter- 
minals are more responsible for responding to it 

Broadband asynchronous transfer mode (ATM) 
networks often have links that span large distances, 
thousands of miles in many cases. Here the pro pa- 45 
gation delay is too long to allow feedback to be effec- 
tive. The delay from the time a packet is sent to the 
time it is received at the destination is much longer 
than the time during which congestion can cause buf- 
fers to fill in intermediate nodes, so data is lost By the so 
time the loss is recognized and a feedback signal sent 
back to the sender, it is too late to alter the sending 
rate or otherwise change the input to prevent conges- 
tion. 

It is not sufficient to use any of the common feed- 55 
back schemes, including credit managers, windows, 
etc., across long-delay ATM networks. While some of 
these techniques are quite appropriate for short-haul 



ATM applications, they lose effectiveness when the 
buffer fill time falls well below the propagation delay 
in the link. The exact point at which feedback delay 
becomes unacceptable depends upon the degree of 
burstiness of the traffic; if the bulk of traffic is constant, 
then a somewhat longer delay can be tolerated before 
loss occurs. Highly bursty traffic is more sensitive. 

In a typical network, therefore, at least two types 
of bandwidth allocation are needed. A simple credit- 
based buffer allocation scheme is likely to be quite 
adequate for certain applications - those links that 
have short propagation delay. The receiving end of 
each link monitors the buffers avaflable to each virtual 
path and/or virtual channel and grants credits to the 
sending end. This may be somewhat cumbersome 
when the number of virtual paths is quite high, but in 
practice credits are allocated back to each virtual path 
or channel based upon availability of buffers for the 
links going out of the node. Some amount of receive 
buffer may be useful in order to permit a node to 
accept all traffic arriving at one incoming link when 
there is a disparate buffer fill situation at its outgoing 
links. Nonetheless, this transmission discipline is sim- 
ply a form of conventional hop-by-hop management 
and is not dis-similar from what is found on conven- 
tional connection-oriented packet networks. These 
links may not need a more complex scheme such as 
is described below. A more complex discipline is only 
required when the dimensions of the network cause 
propagation delays to become longer than allowable 
feedback times. 

There are two causes of congestion loss, funnel- 
ling and mismatch. A packet-switched network can 
lose its protocol data units (cells, frames or packets) 
when the arrival rate at any given point exceeds the 
departure rate for a long enough period of time that a 
buffer overflows. This can occurfor either of two sepa- 
rate and identifiable reasons. Funnels occur when 
several different paths converge on a single buffer, 
and traffic bursts arrive closely spaced in time, such 
that overflow occurs. Funneling is generally transient 
Mismatch occurs when sustained demand for a given 
facility exceeds its capacity; for example, when a 
high-speed link meets a lower-speed link, or when an 
additional virtual circuit is created over a busy facBity. 
A congestion management scheme must be able to 
handle both mismatch and funneling. However, defe- 
rent techniques tend to be more effective for one or 
the other. Admission control policies, coupled with 
stringent network-wide resource allocation and a 
minimum of oversubscription, can minimize mis- 
match. Connectionless networks require feedback to 
control mismatch loss, as they rarely if ever provide 
rate-based control. Traditional packet networks are 
usually rather tolerant of funneling loss. An occasional 
dropped packet can be recovered. ATM networks, 
however, may use protocols that are prone to loss 
multiplication; a single dropped cell can corrupt an 
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entire packet, if frame-based recovery is used. Thus, 
tunneling effects are far more severe in an ATM con- 
text and are hardest to solve. 

It has been suggested that by limiting the rate at 
which users are allowed to send data into an ATM net- 
work (i.e., access control) such that the total 
bandwidth of all channels does not exceed the cross- 
sectional size of any trunk facility, then congestion will 
not occur. This is not true, however, when the traffic 
is bursty. Wh3e bursts in an ATM network may be indi- 
vidually bounded in size and rate, a probability exists 
that at any given time the amount of traffic arriving for 
any given buffer will exceed the capacity of that buffer, 
even if the average is not excessive. 

One cause of funneling effect is most likely when 
many small virtual channels are provided. If the num- 
ber of virtual channels exceeds the number of cells in 
a buffer, then it is statistically possible that all of them 
send their cells at such times that the bursts arrive at 
a given buffer dose enough in time that the buffer 
overflows. The total "event horizon' within an ATM 
network is no greater than the longest round-trip delay 
including buffer times. Thus, even circuits with a 
"reserved" throughput class (enforced at the access) 
of, say, 64- kbps, who are thus allowed to issue one 
cell (with a 48-octet paytoad) every 6-ms., can send 
those cells anywhere within the 6-ms. window, and 
indeed more likely will be allowed to accumulate 
credits so that larger bursts may be sent with less fre- 
quency. 

A second cause of funneling loss is the simul- 
taneous arrival of multiple large bursts of traffic at a 
common point High-speed data traffic, such as 
occurs on local area networks and which may migrate 
to ATM or other wide are: networks, is characterized 
by bursts of data at or near the data rate of the physi- 
cal facility, and consisting of more information than 
might fit into a typical ATM network buffer. Even two 
such bursts of data converging onto one facility may 
result in funneling loss, as the buffer is smaller than 
the bursts and data is arriving more rapidly than it is 
exiting the buffer. 

It can therefore be shown that no access control 
scheme can positively prevent buffer overflow. If ATM 
networks are to use loss- sensitive protocols, then a 
different mechanism is required to prevent cell loss. 
Such a mechanism must actively counter the bursty 
nature of ATM traffic, to reduce peak buffer occu- 
pancy and thus the chance of overflow. Continuous bit 
rate services are in this respect little different from 
variable bit rate services, because typical variable bit 
rate user variations in rate occur over a time period 
that is quite long, compared to sub-millisecond buffer 
fill times. 

Variable bit rate is thus handled by treating it as 
a special case of continuous bit rate, in which the bit 
rate is changed on occasion. Most bursty data can tol- 
erate delays in the 100-ms. range; rf re all oca- tion of 
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bandwidth takes this long, applications and users will 
typically not notice. The invention is thus capable of 
operating over circuit- switched networks with high 
switching speed, as well as over ATM networks. 

5 Circuit-switched digital networks (including nar- 

row bandwidth integrated services digital networks) 
typically make use of synchronous time division multi- 
plexing to allocate bandwidth on trunk facilities; in this 
method, individual channels are separated by their 

10 position within a stream. Some bandwidth is required 
for framing purposes, but individual channels have no 
overhead of their own. Asynchronous transfer mode 
(ATM) is, in effect, "asynchronous time division multh 
plexing", where individual channels are identified by 

is label, instead of by position within the stream. 
Asynchronous transfer mode is thus more akin to 
packet mode in operation, although it operates below 
the data link layer and does not provide the same ser- 
vices as narrowband packet networks. 

20 Because the bursty nature of ATM will neces- 

sarily result in buffer overflow in heavily-loaded (but 
not necessarily oversubscribed) networks with long- 
delay physical facilities, buffer management (i.e., con- 
gestion avoidance) requires a redefinition of the 

25 problem, as addressed by the present invention. 

SUMMARY OF THE INVENTION 

A technique is described herein that is compatible 

30 with asynchronous transfer mode, uses ATM-type 
labeled cells and provides similar services, but is not 
completely asynchronous; this technique is applic- 
able to variable bit rate applications. 

The invention herein described may be use d in 

35 various types of networks; one network which may 
use the features of the invention employs a "plesioc- 
hronous" transfer mode (PTM) (plesiochronous=near 
synchronous) The plesiochronous transfer mode pro- 
vides services like asynchronous transfer mode 

40 (ATM) but with much lower (albert non-zero) probabi- 
lity of cell loss, and is intended for use on long-delay 
links within ATM networks. PTM uses cells like ATM, 
but prevents buffer overflows by pre-all oca ting 
bandwidth. Like synchronous time division multiplex- 

45 ing, time slots are used, but here the slots are one cell 
wide (53-octets) and labeled with cell headers. Slot- 
ting is thus used solely as a buffer management and 
congestion control mechanism. The operation of a 
plesiochronous transfer mode link is similar to that of 

so a phase locked loop, with each pun link operating with 
a fixed frequency (periodicity). Within each PTM link, 
a fixed number of slots, each carrying one ATM cell, 
is provided. Individual virtual channels are assigned 
slots, based upon the required bandwidth. Because 

55 operation is plesiochronous (near-synchronous) and 
not synchronous, buffering is still required. 

The invention in its broad form resides in a 
method of congestion avoidance in a communications 
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network having multiple nodes, comprising the steps 
of. a) sending from a first of said nodes to a first inter- 
mediate one of said nodes a request for an allocation 
of bandwidth for a transmission of a quantity of data 
from said first node to a second one of said nodes via 
said first intermediate node and a second inter- 
mediate node; characterized by: b) comparing said 
request in said first intermediate node with capacity at 
said first intermediate node to meet said request and 
generating a modified request to reduce said allo- 
cation if necessary; c) sending said modified request 
from said first intermediate node to said second inter- 
mediate node; d) comparing said modified request in 
said second intermediate node with capacity at said 
second intermediate node to meet said modified 
request, and generating a second modified request to 
reduce said allocation if necessary; e) sending said 
second modified request back to said first node; and 
f) transmitting said quantity of data from said first node 
to said second node using the bandwidth specified in 
said second modified request 

The invention also consists in a communications 
network having multiple nodes, comprising: a) a trans- 
mitter sending from a first of said nodes to a first inter- 
mediate one of said nodes a request for an allocation 
of bandwidth for a data transmission from said first 
node to a second one of said nodes via said first inter- 
mediate node and a second intermediate node; b) 
means for comparing said request in said first inter- 
mediate node with capacity at said first intermediate 
node to meet said request and for generating a mod- 
ified request to reduce said allocation if necessary, 
said modified request being sent from said first inter- 
mediate node to said second intermediate node; c) 
means for comparing said modified request in said 
second intermediate node with capacity at said sec- 
ond intermediate node to meet said request, and for 
generating a second modified request to reduce said 
allocation if necessary, said second modified request 
being sent back to said first node; d) said transmitter 
sending data from said first node to said second node 
using the bandwidth specified in said second modified 
request. 

In accordance with one embodiment of this inven- 
tion, a data communication network of the type sub- 
ject to bursty traffic and having long-delay links 
employs an asynchronous transfer mode in which 
small, fixed-length blocks of information (cells) are 
transferred at very high speed. The network employs 
a bandwidth allocation scheme to avoid congestion. 
When a source node has a burst of traffic to send, it 
first sends a bandwidth request message through the 
network from source to destination. At each inter- 
mediate node, this bandwidth request is examined 
and the node determines how much of the requested 
traffic level it will be able to support by reservation at 
a time in the future of one round-trip interval hence, 
and this node either grants the request or marks down 
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ttie request to a level that it can support then passes 
it on. When the request reaches the destination, it is 
returned along the same path to the source, and the 
source then employs the marked-down allocation to 
5 select the rate used to send the burst of data. The allo- 
cation for this source node remains in effect for a 
limited time, depending upon the amount of data to be 
sent in the burst then returns to a "residual* level. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

The novel features believed characteristic of the 
invention are set forth in the appended claims. The 
invention itself, however, as well as other features and 
15 advantages thereof, will be best understood by refer- 
ence to the detailed exemplary description of specific 
embodiments which follows, when read in conjunction 
with the accompanying drawings, wherein: 

Figure 1 is an electrical diagram in block form of 
20 a communica- tions network in which one embo- 

diment of the invention may be utilized; 
Figure 2 is an electrical diagram in block form of 
one of the nodes in the network of Figure 1; 
Figure 3 is a diagram of the format of a packet 
25 which may be employed in some links of the net- 

work of Figure 1; 

Figure 4 is a timing diagram of the format of a 
frame or synchronized loop which may be 
employed in a long-delay link of the network of 

$0 Figure 1, according to the PTM technique; 

Figure 5 is a diagram similar to Figure 1 of a part 
of a network having two long-delay links; and 
Figure 6 is a timing diagram of a series of the 
frames of Figure 4 transmitted by one node of Fig- 

35 ure 5 and received by another node; 

Figure 7 is a timing diagram of an allocation 
request message used in the network of Figure 1 
or 5, according to a feature of the invention; and 
Figure 8 is a diagram of message traffic as a func- 

40 tion of time in a system of Figures 1 or 5. 

DETAILED DESCRIPTION OF SPECIFIC 
EMBODIMENT 

45 Referring to Figure 1 . a communications network 

is illustrated having a communications link 10 be- 
tween a pair of nodes 1 1 and 12. The link 10 is a trunk 
facility, where the link 10 must carry traffic between a 
large number of nodes 13 connected to the node 11 

so by links 14 and a large number of nodes 15 connected 
to the node 12 by links 16. In turn, the nodes 13 and 
15 are connected to other nodes 17 and 18. Thus, 
because of the fan-in of a number of nodes to the 
nodes 1 1 and 12, the link 10 must have much greater 

55 capability than the link 19 between nodes 13 and 17, 
for example. The link 1 0 may be a satellite link, orfibre 
optic link, which may span hundreds or thousands of 
miles, and would have a speed of perhaps 600- 

5 
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Mbit/sec or more, i.e., broadband. The links between 
nodes 11, 13 and 17 {or 12, 15 and 18), however, may 
be broadband facilities operating at perhaps 150- or 
600-Mbps, or other speeds, or may be narrowband 
facBities, i.e., 64000-bps. 

The network of Figure 1 may carry both voice and 
data, with the voice messages being in digital format, 
so the network may be of the ISDN or integrated ser- 
vices digital network type. Data channels are charac- 
terized by sporadic bursts of information, and by 
one-way or two- way transmissions at any one time, 
over a wide range of average bandwidth. Thus, a large 
number of channels may be funneled through a 
broadband trunk facility such as the link 10 of Figure 
1 . Voice channels send data at a constant bit rate. 

While the link 10 is broadband, there is neverthe- 
less a limit on the amount of traffic it can handle, so 
some type of control on access by the nodes must be 
imposed. A simple method of control would be to 
determine the number of nodes in the entire network 
at a given time and divide the avalable bandwidth 
equally (introducing priorities if needed). However, 
this method would not take advantage of the statistical 
gain inherent in a shared data path; many or most of 
the nodes would not be using their allotted bandwidth 
at a given moment, so much of the network capacity 
would be sitting idle whQe some nodes needing heavy 
message traffic would be delayed in their operation. 
Oversubscribing to take advantage of the statistical 
gain, however, results in congestion at times of heavy 
traffic, with loss of data due to buffers filling up at inter- 
mediate nodes. 

It has been found advantageous to employ an 
asynchronous transfer mode (ATM) in networks of the 
type seen in Figure 1. That is, the data flow in the fink 

10 is buffered at nodes 11 and 12 and transmissions 
are not necessarily synchronized with the originating 
node 17 or 18. A block of the 64000-bps data from a 
node 17 may be buffered at node 1 1 and sent to node 
1 2 at 150- or 600-Mbps (depending upon the network 
construction) when it is convenient for the node 11 to 
do so (unrelated to the dock used by the node 17 to 
send data to the node 13, or by the node 13 to send 
data to node 11), yet the data rate of the trunk facility 
is so high that no delay is perceptible at the originating 
node 17. 

The time delay inherent in the link 10 may bp too 
long to allow any meaningful feedback from the node 

11 to the node 12 about the receive conditions. If the 
link 10 is a satellite link, this delay would be several 
hundred milliseconds, tf the link 10 is fibre optic, then 
the speed of transmission is about 200km/msec, so 
a link across the continent creates a delay of many 
milliseconds, which is still too long for feedback to be 
effective. If the node 1 1 had a buffer for each of its 
egress ports 14 of a size of 150 ceils, for example, 
where each cell is 424 bits (53 octets) long, and the 
link 10 had a paytoad of 150- Mbps, then if a two-to- 



BNSDOCID: <EP 0453355A2_I_> 



one mismatch occurred at one buffer (one of the ports 
to links 14) this buffer would go from empty to full in 
about (150 X 424)/(1.5 X 105) = 0.424 milliseconds. 
It is thus seen that a feedback scheme for control of 

5 traffic flow would be inadequate when propagation 
time delays measured in milliseconds are prevalent. 
A fundamental rule of control theory is that feedback, 
to be effective, must be received quickly enough so 
that response can be timely; delayed too much, feed- 
to back will not have the desired effect In broadband 
communications networks using asynchronous trans- 
fer mode, as discussed with reference to Figure 1. 
operating over wide area topology, the event being 
controlled (a buffer filling up at a destination or at 

15 some intermediate point) can occur before the feed- 
back (traveling at the speed of light, or at the signal 
speed in the medium in question) can reach the 
source node (the point to be controlled). 

Thus, due to the vast difference in propagation 

20 delays among the links in Figure 1, the plesiochron- 
ous transfer mode as herein described might be used 
for the link 10 which is long-delay, while credit-based 
ATM (Le„ a shutter or loose window") is used for the 
short-delay links 14 or 16, for example. 

25 Referring to Figure 2, a typical construction of one 

of the nodes 11 or 12 is shown, and the nodes 1 3 and 
15 may be similarly constructed. Although the node 
1 1 of Figures 1 or 2 is illustrated to have three links 
14 and one link 10 as the ingress and egress ports, it 

30 is understood that the node 1 1 may have many more 
ports than four. The link 10 has a transmit line 20 and 
a separate receive line 21, and these are connected 
to a transmitter 22 and a receiver 23, respectively, in 
the node 1 1 . Similarly, each one of the links 14 has a 

35 transmit line 24 and a separate receive line 25, and 
again each of these is connected to a transmitter 26 
or receiver 27, respectively. Although it is not neces- 
sarily the case, the link 1 0 in this example is of broader 
bandwidth (higher rate of transmission) than the links 

40 14, so traffic is funneled into link 10 from several links 
14. The function of the receivers 23 or 27 is to detect 
and demodulate the signal on the line 21 or 25, 
recover the clock, convert the serial data on the line 
to parallel data for loading into the receive buffers. 

45 The function of the transmitters 22 or 26 is to move 
data from a buffer in parallel format, convert the data 
from parallel to serial, modulate a carrier with the 
serial data, and send the data signal out on the trans- 
mit line 20 or 24. Each one of the ports for links 14 is 

so operated by a controller 28, and the port for the link 
1 0 is operated by a controller 29. These controllers 
are usually processors executing code stored in local 
memory, or may be state machines, or similar logic. 
A receive buffer 30 is provided for incoming data from 

55 the line 25 for each of the links 14, and likewise incom- 
ing data on the line 21 of the link 10 is buffered in a 
buffer 31. A transmit buffer 32 may also be provided 
for outgoing data on lines 24 for each link 14, as well 

6 
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as a transmit buffer 33 for the outgoing line 20 of the 
link 10. Although shown as separate transmit and 
receive buffers, these functions may be combined. 
The controllers 28 or 29 are responsive to decoded 
command information in signals, cells or packets on 
the incoming lines 25 or 21 to activate the receivers 
27 or 23 to start loading the respective buffers 30 or 
31 with the incoming data, as well as to route the 
received data to be transmitted at one of the other 
ports. As seen in Figure 3, a cell 34 by which infor- 
mation may be conveyed from a node 13 to the node 
11 is illustrated. This cell 34 is delineated by the 
underlying service, or by some element within the 
header (i.e.. the header checksum). The cell begins 
with a header 35 which includes a virtual channel 
identifier 36. a control area 37 and a header check- 
sum 38 used to verify the integrity of the header and 
of the framing. The payload field 39 is the major part 
of the cell 34. The controller 28 for a port to a link 14 
is responsive to the virtual channel identifier 36 to con- 
trol the routing of the incoming cell through the switch- 
ing network 43 to attempt to pass the cell from one 
port to another in order to effect the virtual channel be- 
tween source and destination. When there is a differ- 
ence in bandwidth between the links 14 and the link 
1 0. for example, the switching network 43 may indude 
a multiplexer 44 to allow more than one link 1 4 to fun- 
nel into the link 10; likewise, a multiplexer 45 may 
allow simultaneous delivery of data from link 10 to 
more than one of the links 14. Similarly, the ports for 
links 14 may have multiplexers 46 and 47 so that data 
from or to multiple ports may be interleaved. Alterna- 
tively, the data may be interleaved by merely reading 
and writing between buffers 30-33 one word at a time 
via the switching circuit 43. In any event, a message 
frame is made up in transmit buffer 33. for example, 
by the controller 29, and this frame may contain inter- 
leaved packets or cells from many different terminals 
17. going to many different terminals 18. 

Referring to Figure 4, a message frame 50 used 
in the PTM technique is illustrated. This frame' 50, 
employed for transmission on the link 10 in one 
example, is of fixed length 51 and is made up of a 
large number of slot cells 52. In an exemplary embo- 
diment, the slot cells 52 each contain fifty-three octets 
(424-bits). and there are 2119 cells in a frame 50 of 
6- millisecond length 51 , transmitted at a rate of about 
1 50-Mbps. A slot cell 52 contains a data field 53 of 48- 
octets and a header 54 of five octets; the header 
includes a channel identifying number associated 
with a particular transmission from a source to a des- 
tination node. The first two cells of the frame 50 are 
sync cells 55; these sync cells delimit each frame 50 
which is sent during a loop control period. At least two 
sync cells 55 are sent at the beginning of each frame. 
Sync cells are identified by a specific header address, 
and each contains a pointer to the first slot in the con- 
trol period (i.e.. the first data cell 52 in the frame) 
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which follows sync and slip cells. (The second sync 
cell contains a pointer value of one lower than the first 
sync cell.) At least one slip cell 56 follows the sync 
cells; a slip cell contains no information, other than a 

5 header address identifying it as a slip cell. These slip 
cells exist only to be added or discarded, as required, 
to synchronize the two sides of a loop (in node 1 1 and 
node 12, for example) when they are not running at 
identical speeds. Typically one slip cell 56 is sent after 

10 the sync cell 55, but a second will be added, or the one 
will be deleted, as required. The slot cells 52 are the 
ones assigned to carry slotted traffic, each having a 
valid virtual channel identifier in its header 54. Slot 
cells are carried with priority over free cells, and are 

15 allocated using control cells. A free cell 57 is an unal- 
located cell, and may be empty, in which case its 
header carries a virtual channel identifier for an empty 
cell, or may carry traffic for which no slot is assigned; 
this unallocated traffic is carried on a best-effort basis 

20 and may be discarded in favor of slot celts when allo- 
cated traffic appears. Finally, a control cell 58 is one 
that carries information (control signals, commands, 
etc.) between the two ends of the loop, e.g., from node 
1 1 to node 12. A control cell 58 is identified by a speci- 

25 fic virtual channel identifier in its header 59 (which 
may be locally assigned). Control cells carry mes- 
sages that indicate that a given time slot within the 
basic control period of the frame 50 has been assig- 
ned to carry traffic on behalf of a given virtual channel, 

30 or has been freed. A protocol is defined for sending 
these messages in control cells, and the controller 29 
generates these cells for sending via transmitter 22. 

The frames 50 are timed by a separate clock 60 
in each node 1 1 or 12. These docks are stable crystal 

35 oscillators which maintain sufficient accuracy to 
synchronize the repetition of frames 50 at each end of 
a link 10. Since the time period 51 of a frame 50 is 
some multiple of 6-millisecond. this level of accuracy 
is well within that of currently-available crystal oscil- 

40 lators, or other methods of establishing a stable time 
reference. The bit rate, about 150-Mbps (or 600- 
Mbps, depending upon the network), is established by 
the oscillator 60. and the octet, cell and frame rates 
may be obtained by counting down from the bit rate 

45 clock, or from external synchronization sources, i.e., 
a network master clock. 

Referring to Figure 5, an example of connection 
establishment is illustrated where a seven-hop con- 
nection (including two local loops) is shown between 

so two of the terminals 17 and 18. labelled Y and Z in this 
example. Nodes 11, 11a and 12 are of the type shown 
in Figures 1 and 2, using the framing loops of Figure 
4 in links 10 and 10a the link 10 is assumed to have 
an 11 -millisecond one-way propagation delay, 

55 synchronized at four base periods or 24-ms. with 
8192 slots 52 in a frame 50, while link 10a is assumed 
to have a 2-ms. one-way propagation delay, 
synchronized at one base period of 6-ms with 2048 
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slots. No assumption is made that bandwidth is sym- 
metrical; traffic is to be passed from Y to Z, without 
regard for traffic from Z to Y (which is handled sepa- 
rately). When the connection is requested by terminal 
Y, the network controller first identifies the path from 
Y to 2. determined here to be a path 

Y -> A -> B -> C -> D -> E -> Z. 
Links A-B and D-E, like the local loops Y-A and E-Z, 
are local and can maintain an acceptably low loss rate 
by using conventional local credit management 
techniques. Links B-C and C-D (like link 10 of Figure 
1) are longer and are phase-locked to one another 
and to the network master clock (or local 
synchronized docks 60). Nodes B t C and D thus per- 
form a phase-comparator function upon their plesioc- 
hronous transfer mode links so that a fixed mapping 
between slots in adjacent links is possible, using the 
frames 50 of Figure 4. 

Once the path Y-to-Z is identified, the network 
controller (i.e., one of the controllers 29) allocates the 
appropriate bandwidth along each link. The 
unsynchronized links such as A-B and D-E need 
merely to have sufficient bandwidth available. The 
phase-locked links B-C and C-D have slots 52 allo- 
cated to the virtual channel Y-Z. Slots are assigned as 
far apart within the loop period 51 as possible, in order 
to minimize funneling effects. The number of slots 
allocated in each link is based on the bandwidth 
required. A link whose loop control period is greater 
than the basic control period of 6-milIisecond is 
treated as having multiple instances of the basic con- 
trol period. Thus in the example, if the channel Y Z 
requires a bandwidth of 256-kbps, then four slots are 
assigned in link C-D (which is operating at frame 
period of 6-ms.), each slot ideally following 512-siots 
after the previous one. The link B-C (which is operat- 
ing at frame period of 24-ms.). however, requires six- 
teen slots assigned, again ideally 512-slots apart. If a 
link operated at a higher speed, the spacing between 
slots would remain uniform in time, and scaled in num- 
ber of slots per frame. For example, if B-C were a 620 
Mbps link, then it would have 32768 slots, and the six- 
teen slots in half-circuit Y Z would be spaced 2048 
slots apart. 

After the slots are allocated by the network con- 
troller, the network controller signals to Y by a control 
packet or cell that it is ready to accept traffic. Access 
node A grants credits to Y. and in turn forwards the 
cells it receives from Y on to B when it receives suffi- 
cient credits from B. B in turn inserts the cells into the 
appropriate time slots in the frame 50 currently being 
sent by B onto link B-C; C does the same in relaying 
them to D. Link D-E, however, is controlled by a sim- 
ple credit mechanism so D buffers the cells until E has 
granted the required credits, at which time it forwards 
them to E, who in turn relays them asynchronously to 
Z. 

Referring to Figure 6, a data stream sent on link 
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10a from a node 11a, for example, to the node 12 is 
a series of frames 50a, 50b, 50c, et seq., where each 
frame is of the format of Figure 4. The actual propa- 
gation delay 61 in the link 1 0a between nodes 11a and 

5 1 2 is less than the length 51 of each of the frames 50a, 
50b, etc. The receiver in the node 12 is synchronized 
by its dock 60 to receive the frame 50a beginning at 
time 62, and subsequent frames at times 63 spaced 
by the period 51, in a continuous sequence. If the 

10 clocks 60 drift, or the propagation delay drifts due to 
environmental factors, then there is some elasticity in 
the receive data buffers to account for such differ- 
ences, while a drift of a cell length is accounted for 
adding or deleting slip cells 56. Drift of the order of 

is magnitude of the bit rate (150- or 600-Mbps) or at the 
octet rate (1 8.75- or 75-M/sec.) is accounted for in the 
elasticity of the receive circuitry. The "phase- locking" 
referred to between transmitted and received frames 
in at slot level and frame level. Note that the slots or 

20 cells are phase locked between the various links in 
the network, and the frames are phase locked be- 
tween a transmit-receive pair of nodes, but the frames 
50 may be of different length in link 10 compared to 
frames in link 10a. 

25 Significantly, and as described herein, a different 

allocation method is used for bursty traffic. This sec- 
ondary procedure, used only for bursty (variable bit 
rate) traffic, employs a form of "fast circuit" switching, 
in which virtual channels are varied in size in order to 

30 handle variations in offered load. Access nodes or ter- 
minals such as node A of Figure 5 are expected to buf- 
fer traffic as it arrives, but when the buffer begins to 
fill, this originator node A may request a temporary 
increase in bandwidth sufficient to empty itself. 

35 Alt virtual channels such as the channel Y Z have 

a residual bandwidth (BRes) which is available at all 
times; in the above example the residual bandwidth 
may be one slot per 512 slots, to be automatical- ly 
allocated by the network controller whenever a 

40 request is made, without any exchange of signals be- 
tween nodes 11 and 12. Additional bandwidth for a 
given virtual channel is requested by means of a 
bandwidth request descriptor which is a message 
packet or cell 64 as seen in Figure 7, sent from the 

45 node 13 to the node 11, for example. This request 
descriptor message of course includes a field 65 to 
identify the source node and the destination node (or 
channel number) so that the path can be determined, 
and in addition has three elements, a BWext field 66, 

so a BWquo field 67 and a duration field 68. The BWext 
field 66 specifies the requested bandwidth extension, 
and is a value representing the most that the network 
will grant for the duration of the descriptor. The net- 
work control facBity determines what maximum 

55 bandwidth will ever be allotted to a terminal, depend- 
ing upon the network configuration at the time, and 
sends this value to alt nodes. The BWext field 66 is in 
the message 64 when it is sent by the originator to the 
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network. The BWquo field 67 is the bandwidth exten- ~ 
ston quota, which is the amount that the network actu- 
ally grants. This value is initially set by the originating 
terminal 13 to be equal to BWext but may be lowered 
by any of the intermediate nodes 1 1. 1 1a. 12. etc.. bef- 5 
ore being returned to the originator by the network. 
The duration field 68 is the amount of time that the 
descriptor should remain in effect which is ideally 
expressed as a number of cells. While a time-based 
descriptor could be used, it would have to be exten- 10 
ded by the network if the request BWext were reduced 
to a lower BWquo. since it would take longer to send 
the pending traffic. 

Every node within a network such as that of Fig- 
ure 5 also determines, at connection establishment, 15 
the total end-to-end transit delay across the network 
and its own relative position within the link (i.e., how 
many ms. from each end). When a message or 
bandwidth request descriptor or cell 64 is issued, this 
procedure is followed: (1.) The originator (e.g.. node 20 
13) sends a bandwidth descriptor 64 across the link 
towards the destination terminal Z. This cell 64 is iden- 
tified as a user-to-network cell by its header 69. (2.) 
As the cell 64 travels towards the destination (using 
the transport mechanism of Figure 5. for example), 25 
each node 11. 11a 12, etc., determines how much of 
the requested bandwidth in BWext it can provide 
exactly one round-trip interval hence. If it cannot pro- 
vide at least as much as the current BWquo in field 67 
(which is being marked down as the descriptor 64 30 
travels towards the destination), it puts a new value in 
BWquo field 67. No node may raise the value in 
BWquo field 67 as the cell 64 traverses the network. 
(3.) At the egress node (E in Figure 5), the descriptor 
64 is returned along the same path by which it arrived. 35 
Each node (12. 1 1a, 1 1 , etc) on the return path notes 
the remaining (marked down) value of the BWquo 
field 67, but does not further change it; this value is 
stored in a table of all current traffic, identified by the 
channel number in the field 65. so that the controller 40 
29 can check subsequent slotted cells for validity and 
also keep track of allocated capacity when making 
allocation for subsequent requests 64. (4.) When the 
descriptor 64 returns to its originator node 13, the 
bandwidth described in BWquo field 67 becomes 45 
available for immediate use by the terminal Y. for the 
time period of the duration field 68. 

The concept of a virtual path instead of a virtual 
channel may be used to minimize node complexity. 
The total number of virtual channels between any two so 
nodes such as Y and 2 in a network is likely to fre- 
quently exceed one, e.g., when more than one mes- 
sage is pending. Some economization may take place 
by allocating bandwidth descriptors 64 to virtual paths 
instead of virtual channels. There can be only one vir- ss 
tual path between two nodes Y and Z. A virtual path, 
in this case, is a special form of channel that contains 
within itself multiple user virtual channels. The total 
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number of virtual paths within a network is thus limited 
to the square of the number of nodes, regardless of 
the number of virtual channels requested by users. 
Access nodes can map each virtual channel into the 
appropriate virtual path, and intermediate nodes need 
only keep track of virtual paths (as well as any virtual 
channels that are locally terminated). 

Referring to Figure 8. a chart of the allocated traf- 
fic in one link 10, for example, as a function of tine, 
shows that as the number of requested allocations 
from the remote nodes changes the allocated traffic 
level follows a line 71, rising and falling as the 
requests ebb and flow. A line 72 represents the limit 
imposed by the capacity of the link, determined by the 
physical construction, software, etc. During a peak in 
traffic, when the requested allocations from the 
remote terminals tend to exceed the limit 72 to follow 
the line 73. the controller imposes reduced (instead of 
requested) allocations on all remotes so the real traffic 
follows the line 74, below the limit 72, instead of the 
line 73. All network traffic is at a lower level than 
requested for a time period 75. until the line 76 rejoins 
the request curve 77 when all delayed requests have 
been made up. In this manner, network congestion 
during the peak period 75 is merely exhibited to the 
remote terminals as a slowing of the apparent res- 
ponse of the network, rather than as loss of data 
requiring retransmitting sequences of messages. 
Retransmission occurrences not only markedly 
reduce the apparent speed of the network from the 
terminals standpoint but also reduce the real capacity 
of the network since traffic is transmitted more than 
once. 

While this invention has been described with 
reference to specific embodiments, this description is 
not meant to be construed in a limiting sense. Various 
modifications of the disclosed embodiments, as well 
as other embodiments of the invention, will be appa- 
rent to persons skilled in the art upon reference to this 
description. It is therefore contemplated that the 
appended claims will cover any such modifications or 
embodiments as fall within the true scope of the inven- 
tion. 



Claims 

1. A method of congestion avoidance in a communi- 
cations network having multiple nodes, (11, 11a. 
12) comprising the steps of: 

a) sending from a first of said nodes to a first 
intermediate one of said nodes a request 64) 
for an allocation of bandwidth for a transmis- 
sion of a quantity of data from said first node 
to a second one of said nodes via said first 
intermediate node and a second intermediate 
node; characterized by. 

b) comparing (34) said request in said first 
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meet said request, and for generating a sec- 
ond modified request to reduce said allocation 
if necessary, said second modified request 
being sent back to said first node; 
d) said transmitter sending data from said first 
node to said second node using the bandwidth 
specified in said second modified request. 

6. A network according to claim 5 wherein said 
means for comparing said request in said first 
intermediate node includes means for comparing 
with capacity projected at a time about that 
required for one round trip between said first node 
and said second node. 

7. A network according to daim 5 or 6 wherein said 
first node ts allocated a bandwidth value [a] equal 
to a standard allocation and only sends said 
request if the bandwidth exceeds said standard 
bandwidth value. 



intermediate node with capacity at said first 
intermediate node to meet said request, and 
generating a modified request to reduce said 
allocation rf necessary; 

c) sending said modified request from said 5 
first intermediate node to said second inter- 
mediate node; 

d) comparing said modified request in said 
second intermediate node with capacity at 
said second intermediate node to meet said to 
modified request, and generating a second 
modified request to reduce said allocation if 
necessary; 

e) sending said second modified request back 

to said first node; and is 

f) transmitting said quantity of data from said 
first node to said second node using the 
bandwidth specified in said second modified 
request 

20 

2. A method according to claim 1 wherein said step 
of comparing said request in said first inter- 
mediate node includes comparing with capacity 
projected at a time about that required for one 
round trip between said first node and said sec- 25 
ond node. 



3. A method according to claim 1 or 2 wherein said 
first node is allocated a standard bandwidth value 
and only sends said request if the bandwidth 
needed for transmitting said quantity of data 
exceeds said standard bandwidth value. 



8. A network according to any of claims 5 to 7 includ- 
ing a transceiver in said first intermediate node 
which transmits data from said first intermediate 
node to said second intermediate node by an 
asynchronous transfer mode. 

9. A network according to any of claims 5 to 8 whe- 
rein said transmitter sends data by a serial data 

30 link. 

1 0. A method of transmitting a quantity of digital infor- 
mation from a node in a network to a destination, 
comprising the steps of: 

a) transmitting from said first node a control 
cell containing a request for a bandwidth allo- 
cation needed for said quantity of information, 
a bandwidth quote equal to said bandwidth 
allocation, and a duration for the requested 
bandwidth; 

b) receiving said control cell at an inter- 
mediate node in said network and marking 
down said bandwidth quote in said immediate 
node in response to the capacity of said inter- 
mediate node at a time when said quantity of 
information is to be transmitted, then transmit- 
ting said control cell toward said destination; 

c) subsequently receiving at said first node a 
return of said control cell in which said 
bandwidth quote is modified downward in 
accordance with capacity of other nodes in 
said network; 

d) transmitting from said first node to said des- 
tination said quantity of information at a 
bandwidth corresponding to said modified 
bandwidth quote. 

11. A method according to claim 10 wherein said 



4. A method according to any of claims 1 to 3 whe- 
rein data is transmitted from said first inter- 35 
mediate node to said second intermediate node 
by an asynchronous transfer mode. 

5. A communications network having multiple nodes 
(11, 11a, 12, 13. 15). comprising: *o 

a) a transmitter (22) sending from a first of said 
nodes to a first intermediate one of said nodes 
a request for an allocation of bandwidth for a 
data transmission from said first node to a sec- 
ond one of said nodes via said first inter- 45 
mediate node and a second intermediate 
node; 

b) means (34) for comparing said request in 
said first intermediate node with capacity at 
said first intermediate node to meet said so 
request, and for generating a modified request 
to reduce said allocation if necessary, said 
modified request being sent from said first 
intermediate node to said second inter- 
mediate node; ss 

c) means for comparing said modified request 
in said second intermediate node with 
capacity at said second intermediate node to 
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duration is expressed in quantity of digital infor- » . r 
mation. 

12. A method according to daim 10 or 11 wherein 
said node is granted a residual allocation of s 
bandwidth, and said control cell is transmitted 
only if said request exceeds said residual 
bandwidth. 

13. A method according to any of claims 10 to 12 10 
including the step of receiving said control cell at 

an intermediate node and marking down said 
bandwidth quote in said control cell at said inter- 
mediate node in response to the capacity of said 
intermediate node at a time when said quantity of is 
information is to be transmitted. 

14. A method according to any of claims 10 to 13 
wherein said information is transmitted from said 

at least one node in said network to another node 20 
by an asynchronous transfer node. 

15. Apparatus for transmitting a quantity of digital 
information from a node in a network to a desti- 
nation, comprising: 25 

a) means for transmitting from said node a 
control cell containing (1) a request for a 
bandwidth allocation needed for said quantity 
of information, (2) a bandwidth quote equal to 

said bandwidth allocation, and (3) a duration 30 
for the requested bandwidth; 

b) means for receiving said control cell at an 
intermediate node in said network and modify- 
ing downward said bandwidth quote in said 
control cell in response to the capacity of said 35 
intermediate node at a time when said quan- 
tity of information is to be transmitted; 

c) means for transmitting said control cell 
including said modified downward bandwidth 
quote from said intermediate node toward 40 
said destination; 

d) means in said first node for subsequently 
receiving at said node a return of said control 
cell in which said bandwidth quote is modified 
downward in accordance with capacity of 45 
other nodes in said network; 

e) and means in said node for transmitting 
from said node to said destination said quan- 
tity of information at a bandwidth correspond- 
ing to said modified downward bandwidth so 
quote. 
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